Development and Deployment of a Large-Scale Flower Recognition Mobile App

Today’s major image search engines, which focus on searchby-image-content, work by matching and retrieving of images that are already available on the web. With the proliferation of user generated content, especially from mobile devices, there is a need to develop applications which are more content-aware, i.e. can understand and recognize what is in the image, and which can handle the deteriorated quality inherent to user images uploaded from mobile devices. In this paper we describe a mobile phone application intended to automatically recognize flower images taken by users and classify them into a set of flower categories. The app is served by a web-scale image search engine and relies on computer vision recognition technology. The mobile phone app is available free to users, and as a web interface. We share experiences about designing, developing, and deploying such a system in practice. We discuss practical aspects of applying the object recognition technology, including runtime, scalability, and data collection which is crucial for such data-driven application. More specifically, we describe a large-scale dataset which was collected in a number of stages to be as close as possible to the data distribution the actual users of the app will encounter. We further describe our strategy for collecting user feedback, which we view as an opportunity to improve the server-side algorithms and datasets. We envision that these issues will be encountered in similar applications, e.g. for recognition of plants, mushrooms, or bird species, all of which have practical importance for users, and we hope this work will be also useful for developing other types of applications. To our knowledge, this is the first mobile app which can automatically recognize as many as 578 species of flowers, and which is available for free to users. The flower dataset that serves the application, specifically collected for this recognition task, is the first and largest in its scale and scope.